System and method for interactive projection

ABSTRACT

An interactive projection system and method comprising a camera/projector unit having a computer processor connected via a network to a content server and content database. The system projects interactive trigger areas on a three-dimensional object. Specific content stored on the content server or locally in the memory of the computer processor is associated with each trigger area. A user interacts with the trigger areas and the system projects informational or entertainment content about the object on the surface of the object.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 62/097,769 filed on Dec. 30, 2014, the contents of whichare incorporated herein by reference.

FIELD OF THE INVENTION

The disclosure relates to systems and methods for projecting images. Inparticular, the disclosure relates to a system and method for projectionof interactive areas and mapping of the interactive areas onto complexsurfaces.

BACKGROUND OF THE INVENTION

Active product displays are useful in demonstrating advanced features ofvarious products. Prior art systems are typically passive displaysystems, in that content is presented to a viewer on a screen withlittle opportunity for feedback. In other prior art systems, a touchscreen computer is employed to allow the viewer to select content forreview and to provide feedback. However, these systems require extensivecustom set-up or are prohibitively expensive. Further, these systemsrequire extensive maintenance due to frequent wear on the touch screen.

For example, U.S. Pat. No. 7,598,942 to Underkoffler, et al. discloses agestural interface to visually presented elements on a display screen.Multiple cameras capture movement and detect location and orientationand generate output signals to processors connected to a computer. Theprocessors translate the camera outputs into gestural signals which areinterpreted as input information. The computer uses the inputinformation to generate commands to control computer functions.

U.S. Pat. No. 7,034,807 to Maggioni discloses a system for interactionwith a display. The system includes a recording unit for recording apointer object in the vicinity of the display. A computer is connectedto the recording unit and is used to determine a position of the pointerobject. The system requires multiple cameras and multiple reflectivesurfaces.

U.S. Patent Publication No. 2012/0212413 to Plagemann, et al. disclosesa system for receiving image information and translating it intocomputer instructions. Image information is received for a singlepredetermined action space to identify motion. A camera combined withmirrors, prisms, or optic cables is used to gather the imageinformation. However, only one action area is monitored at any giventime.

U.S. Patent Publication No. 2012/0162077 to Sze, et al. discloses aninput device used to detect locations and motions of objects in avirtual working area. A camera is directed to a region of interest. Aregion of interest is illuminated by a “flat” beam of light. An objectis placed in the region of interest and illuminated. The camera capturesan image of the object and sends it to the processor. The processorprocesses the image to obtain locations and movements. Based on themovements, the processor produces computer instructions.

Therefore, there is a need for a system and method for projecting aninteractive environment for a display. The system should provide optionsfor the user to actively select the content to view. The system shouldbe easy to set up and should be capable of being easily transported toother locales.

SUMMARY OF THE INVENTION

The system disclosed projects a “touchable” image on a three-dimensionalobject which enables user interaction. The touchable image is defined bya trigger area and a content area.

The system includes an infrared camera and a projector connected to acomputer processor. In one embodiment, a content server and contentdatabase are also connected to the computer via a network. Content to beprojected is stored in the database. During set-up, the system capturesa baseline image of a three-dimensional target surface. The system thentransforms the content to correct for projector position and for targetcurvature. During operation, the corrected content image is projectedonto the three-dimensional target surface along with similarlytransformed image of the trigger area. The infrared camera then monitorsthe trigger area for a trigger event and reports it to the server whenit occurs. A trigger event occurs when an object is placed inside thetrigger area for a predetermined period of time. Upon report of atrigger event, the system fetches the additional stored content,transforms it, and projects it onto the target surface. The computerprocessor uses both affine and a non-affine transforms to translate and“warp” the content image to match the surface of the target object.

In an alternate embodiment, the system uses facial recognition softwareto project predefined content associated with a particular recognizeduser on the surface of the object.

The action areas can be comprised of a plurality of separately definedactions areas each corresponding to different content. When a userinteracts with an action area, the system recognizes the presence of auser and projects the associated content. The affine transformations canbe described through linear algebra. In linear algebra, lineartransformations can be represented by matrices. If T is a lineartransformation mapping R^(n) to R^(m) and {right arrow over (x)} is acolumn vector with n entries, then T({right arrow over (x)})=A{rightarrow over (x)}

For some m×n matrix A, called the transformation matrix of T. There arealternative expressions of transformation matrices involving row vectorsthat are preferred by some authors.

Matrices allow arbitrary linear transformations to be represented in aconsistent format, suitable for computation. This also allowstransformations to be concatenated easily (by multiplying theirmatrices).

Linear transformations are not the only ones that can be represented bymatrices. Some transformations that are non-linear on a n-dimensionalEuclidean space R^(n), can be represented as linear transformations onthe n+1-dimensional space R^(n+1). These include both affinetransformations (such as translation) and projective transformations.For this reason, 4×4 transformation matrices are widely used in 3Dcomputer graphics. These n+1-dimensional transformation matrices arecalled, depending on their application, affine transformation matrices,projective transformation matrices, or more generally non-lineartransformation matrices. With respect to an n-dimensional matrix, ann+1-dimensional matrix can be described as an augmented matrix.

In the physical sciences, an active transformation is one which actuallychanges the physical position of a system, and makes sense even in theabsence of a coordinate system whereas a passive transformation is achange in the coordinate description of the physical system (change ofbasis). The distinction between active and passive transformations isimportant. By default, by transformation, mathematicians usually meanactive transformations, while physicists could mean either.

Put differently, a passive transformation refers to description of thesame object as viewed from two different coordinate frames.

If one has a linear transformation T(x) in functional form, it is easyto determine the transformation matrix A by transforming each of thevectors of the standard basis by T, then inserting the result into thecolumns of a matrix. In other words,

A=[T({right arrow over (e)}₁) T({right arrow over (e)}₂) . . . T({rightarrow over (e)}_(n))]  Eq. 1

For example, the function T(x)=5x is a linear transformation. Applyingthe above process (suppose that n=2 in this case) reveals that

$\begin{matrix}{{T\; \left( \overset{\rightarrow}{x} \right)} = {{5\overset{\rightarrow}{x}} = {{5I\overset{\rightarrow}{x}} = {\begin{bmatrix}5 & 0 \\0 & 5\end{bmatrix}\overset{\rightarrow}{x}}}}} & {{Eq}.\mspace{14mu} 2}\end{matrix}$

It must be noted that the matrix representation of vectors and operatorsdepends on the chosen basis; a similar matrix will result from analternate basis. Nevertheless, the method to find the components remainsthe same.

Mathematically, vector v can be represented in basis vectors, E=[{rightarrow over (e)}₁{right arrow over (e)}₂ . . . {right arrow over(e)}_(n)] with coordinates [v]_(E)=[v₁v₂ . . . v_(n)]^(T):

{right arrow over (v)}=v ₁ {right arrow over (e)} ₁ +v ₂ {right arrowover (e)} ₂ + . . . +v _(n) {right arrow over (e)} _(n) =Σv _(i) {rightarrow over (e)} _(i) =E[v] _(E)   Eq. 3

Now, express the result of the transformation matrix A upon {right arrowover (v)}, in the given basis:

$\begin{matrix}\begin{matrix}{{A\left( \overset{\rightarrow}{v} \right)} = {{A\left( {\Sigma \; v_{i}{\overset{\rightarrow}{e}}_{i}} \right)} = {{\Sigma \; v_{i}{A\left( {\overset{\rightarrow}{e}}_{i} \right)}} = {\left\lbrack {{A\left( {\overset{\rightarrow}{e}}_{1} \right)}{A\left( {\overset{\rightarrow}{e}}_{2} \right)}\mspace{14mu} \ldots \mspace{14mu} {A\left( {\overset{\rightarrow}{e}}_{n} \right)}} \right\rbrack \lbrack v\rbrack}_{E}}}} \\{= {{A \cdot \lbrack v\rbrack_{E}} = {{\left\lbrack {{\overset{\rightarrow}{e}}_{1}{\overset{\rightarrow}{e}}_{2}\mspace{14mu} \ldots \mspace{14mu} {\overset{\rightarrow}{e}}_{n}} \right\rbrack \begin{bmatrix}a_{1,1} & a_{1,2} & \ldots & a_{1,n} \\a_{2,1} & a_{2,2} & \ldots & a_{2,n} \\\vdots & \vdots & \ddots & \vdots \\a_{n,1} & a_{n,2} & \ldots & a_{n,n}\end{bmatrix}}\begin{bmatrix}v_{1} \\v_{2} \\\vdots \\v_{n}\end{bmatrix}}}}\end{matrix} & {{Eq}.\mspace{14mu} 4}\end{matrix}$

The a_(i,j) elements of matrix A are determined for a given basis E byapplying A to every {right arrow over (e)}_(j)=[00 . . . (v_(j)=1) . . .0]^(T), and observing the response vector A{right arrow over(e)}_(j)=a_(1,j){right arrow over (e)}₁+a_(2,j){right arrow over (e)}₂+. . . +a_(n,j){right arrow over (e)}_(n)=Σa_(i,j){right arrow over(e)}_(i). This equation defines the elements, a_(i,j), of j-th column ofthe matrix A.

To represent affine transformations with matrices, we can usehomogeneous coordinates. This means representing a 2-vector (x, y) as a3-vector (x, y, 1), and similarly for higher dimensions. Using thissystem, translation can be expressed with matrix multiplication. Thefunctional form x′=x+t_(x); y′=y+t_(y) becomes:

$\begin{matrix}{\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix} = {\begin{bmatrix}1 & 0 & t_{x} \\0 & 1 & t_{y} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}x \\y \\1\end{bmatrix}}} & {{Eq}.\mspace{14mu} 5}\end{matrix}$

All ordinary linear transformations are included in the set of affinetransformations, and can be described as a simplified form of affinetransformations. Therefore, any linear transformation can also berepresented by a general transformation matrix. The latter is obtainedby expanding the corresponding linear transformation matrix by one rowand column, filling the extra space with zeros except for thelower-right corner, which must be set to 1. For example, the clockwiserotation matrix from above becomes:

$\begin{matrix}\begin{bmatrix}{\cos \; \theta} & {\sin \; \theta} & 0 \\{{- \sin}\; \theta} & {\cos \; \theta} & 0 \\0 & 0 & 1\end{bmatrix} & {{Eq}.\mspace{14mu} 6}\end{matrix}$

Using transformation matrices containing homogeneous coordinates,translations can be seamlessly intermixed with all other types oftransformations. The reason is that the real plane is mapped to the w=1plane in real projective space, and so translation in real Euclideanspace can be represented as a shear in real projective space. Although atranslation is a non-linear transformation in a 2-D or 3-D Euclideanspace described by Cartesian coordinates, it becomes, in a 3-D or 4-Dprojective space described by homogeneous coordinates, a simple lineartransformation (a shear).

More affine transformations can be obtained by composition of two ormore affine transformations. For example, given a translation T′ withvector (t′_(x), t′_(y)), a rotation R by an angle θ counter-clockwise, ascaling S with factors (s_(x), s_(y))and a translation T of vector(t_(x), t_(y)), the result M of T′RST is:

$\begin{matrix}\begin{bmatrix}{s_{x}\cos \; \theta} & {{- s_{y}}\sin \; \theta} & {{t_{x}s_{x}\cos \; \theta} - {t_{y}s_{y}\sin \; \theta} + t_{x}^{\prime}} \\{s_{x}\sin \; \theta} & {s_{y}\cos \; \theta} & {{t_{x}s_{x}\sin \; \theta} + {t_{y}s_{y}\cos \; \theta} + t_{y}^{\prime}} \\0 & 0 & 1\end{bmatrix} & {{Eq}.\mspace{14mu} 7}\end{matrix}$

When using affine transformations, the homogeneous component of acoordinate vector (normally called w) will never be altered. One cantherefore safely assume that it is always 1 and ignore it. However, thisis not true when using perspective projections.

Another type of transformation, of particular importance to thisdisclosure, is the perspective projection. Whereas parallel projectionsare used to project points onto the image plane along parallel lines,the perspective projection projects points onto the image plane alonglines that emanate from a single point, called the center of projection.This means that an object has a smaller projection when it is far awayfrom the center of projection and a larger projection when it is closer.This transformation is used to correct for the position of the projectorrelative to the three-dimensional target object.

The simplest perspective projection uses the origin as the center ofprojection, and z=1 as the image plane. The functional form of thistransformation is then x′=x/z; y′=y/z. We can express this inhomogeneous coordinates as:

$\begin{matrix}{\begin{bmatrix}x_{c} \\y_{c} \\z_{c} \\w_{c}\end{bmatrix} = {\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 1 & 0\end{bmatrix}\begin{bmatrix}x \\y \\z \\w\end{bmatrix}}} & {{Eq}.\mspace{14mu} 8}\end{matrix}$

After carrying out the matrix multiplication, the homogeneous componentw_(c) will, in general, not be equal to 1. Therefore, to map back intothe real plane we must perform the homogeneous divide or perspectivedivide by dividing each component by w_(c):

$\begin{matrix}{\begin{bmatrix}x^{\prime} \\y^{\prime} \\z^{\prime} \\1\end{bmatrix} = {\frac{1}{w_{c}}\begin{bmatrix}x_{c} \\y_{c} \\z_{c} \\w_{c}\end{bmatrix}}} & {{Eq}.\mspace{14mu} 9}\end{matrix}$

More complicated perspective projections can be composed by combiningthis one with rotations, scales, translations, and shears to move theimage plane and center of projection wherever they are desired.

Non-affine transformations such as transformation of a two-dimensionalsurface to a three-dimensional surface can be accomplished, in theory,by treating the three-dimensional surface as a distortion or, in otherwords, a deviation from true rectilinear projection. This type oftransformation is used to correct for the complex shape of thethree-dimensional target object.

Although distortion can be irregular or follow many patterns, the mostcommonly encountered distortions are radially symmetric, orapproximately so. Radial distortions can usually be classified as either“barrel” distortions or “pincushion” distortions.

In barrel distortion, image magnification decreases with distance fromthe optical axis. The apparent effect is that of an image which has beenmapped around a sphere (or barrel). Fisheye lenses, which takehemispherical views, utilize this type of distortion as a way to map aninfinitely wide object plane into a finite image area. In a zoom lensbarrel distortion appears in the middle of the lens's focal length rangeand is worst at the wide-angle end of the range.

In pincushion distortion, image magnification increases with thedistance from the optical axis. The visible effect is that lines that donot go through the center of the image are bowed inwards, towards thecenter of the image, like a pincushion.

A mixture of both types, sometimes referred to as “mustache” distortionor complex distortion, is less common but not rare. It starts out asbarrel distortion close to the image center and gradually turns intopincushion distortion towards the image periphery, making horizontallines in the top half of the frame look like a handlebar mustache.

These distortions are radial distortions. They imply a nonlinear radiusmapping from the object to the image. For example, what is seeminglypincushion distortion is simply an exaggerated radius mapping for largeradii in comparison with small radii. Conversely, barrel distortion issimply a diminished radius mapping for large radii in comparison withsmall radii.

Radial distortion can be corrected using Brown's distortion model, alsoknown as the Brown-Conrady model. The Brown-Conrady model corrects bothfor radial distortion and for tangential distortion caused by physicalelements in a lens not being perfectly aligned. The latter is also knownas decentering distortion.

x _(d) =x _(u)(1+K₁ r ² +K ₂ r ⁴+ . . . +(P ₂(r ²+2x _(u) ²)+2P₁ x _(u)y _(u))(1+P ₃ r ² +P ₄ r ⁴ + . . . ) Eq. 10

y _(d) =y _(u)(1+K ₁ r ² +K ₂ r ⁴+ . . . )+(P ₁(r ²+2y _(u) ²)+2P ₂ x_(u) y _(u))(1+P ₃ r ² +P ₄ r ⁴+. . . )   Eq. 11

where:

(x_(d), y_(d))=distorted image point as projected on image plane usingspecified lens,

(x_(u), y_(u))=undistorted image point as projected by an ideal pin-holecamera,

(x_(c), y_(c))=distortion center (assumed to be the principal point),

K_(n)=n^(th)=radial distortion coefficient,

P_(n)=n^(th) tangential distortion coefficient,

r=√{square root over ((x_(u)−x_(c))²+(y_(u)−y_(c))²)}, and

. . . =an infinite series.

In our case, barrel and pincushion distortions are quadratic, meaningthey increase as the square of distance from the center. In mustachedistortion the quartic (degree 4) term is significant: in the center,the degree 2 barrel distortion is dominant, while at the edge the degree4 distortion in the pincushion direction dominates. Other distortionsare in principle possible—pincushion in center and barrel at the edge,or higher order distortions (degree 6, degree 8)—but do not generallyoccur in practical cases, and higher order distortions are smallrelative to the main barrel and pincushion effects.

Barrel distortion typically will have a negative term for K₁ whereaspincushion distortion will have a positive value. Moustache distortionwill have a non-monotonic radial geometric series where for some r thesequence will change sign.

BRIEF DESCRIPTION OF DRAWINGS

Reference will now be made to the following drawings:

FIG. 1 is a system architecture diagram of a preferred embodiment.

FIG. 2A is a schematic of a camera/projection unit of a preferredembodiment.

FIG. 2B is a schematic of a camera/projection unit of a preferredembodiment.

FIG. 3 is a plan view of a display of a preferred embodiment.

FIG. 4 is a plan view of a display of a preferred embodiment.

FIG. 5 is a top view of a display of a preferred embodiment.

FIG. 6 is a plan view of a display of a preferred embodiment.

FIG. 7 is a plan view of a display of a preferred embodiment.

FIG. 8 is a plan view of a display of a preferred embodiment.

FIG. 9 is a plan view of a display of a preferred embodiment.

FIG. 10 is a plan view of a display of a preferred embodiment.

FIG. 11 is a flow chart of a calibration method of a preferredembodiment.

FIG. 12 is a flow chart of a run time process of a preferred embodiment.

FIG. 13 is a flow chart of a run time process for projection mapping ofa preferred embodiment.

FIG. 14 is a flow chart of a facial recognition process of a preferredembodiment.

FIG. 15 is a system architecture diagram of a preferred embodiment.

FIG. 16A is an elevation view of a preferred embodiment.

FIG. 16B is an elevation view of a preferred embodiment.

FIG. 16C is an isometric view of a hypothetical flat projection surfaceof a preferred embodiment.

FIG. 16D is an isometric view of a curved projection surface of apreferred embodiment.

FIG. 17 is an isometric view of trigger areas of a preferred embodiment.

FIG. 18A is a diagram of a touchable area application in accordance withone embodiment of the disclosure.

FIG. 18B is a diagram of a trigger data structure in accordance with oneembodiment of the disclosure.

FIG. 18C is a diagram of a content data structure in accordance with oneembodiment of the disclosure.

FIG. 19A is a data flow diagram of a system using trigger areas inaccordance with one embodiment of the disclosure.

FIG. 19B is a flow chart for updating a trigger area in accordance withone embodiment of the disclosure.

FIG. 19C is a view of an area of a trigger area in accordance with oneembodiment of the disclosure.

FIG. 19D is a view of a volume of a trigger area in accordance with oneembodiment of the disclosure.

FIG. 19E is a flow chart for updating content in accordance with oneembodiment of the disclosure.

FIG. 19F shows the matrix transformations applied to content inaccordance with one embodiment of the disclosure.

FIG. 20A is a data flow diagram of a system using trigger areas inaccordance with one embodiment of the disclosure.

FIG. 20B is a flow chart for determining a touch of a trigger area inaccordance with one embodiment of the disclosure.

FIG. 21A is a view of the application when a first tab is selected inaccordance with one embodiment of the disclosure.

FIG. 21B is a view of the application when a second tab is selected inaccordance with one embodiment of the disclosure.

DETAILED DESCRIPTION

In the descriptions that follow, like parts are marked throughout thespecification and drawings with the same numerals, respectively. Thedrawing figures are not necessarily drawn to scale and certain figuresmay be shown in exaggerated or generalized form in the interest ofclarity and conciseness.

It will be appreciated by those skilled in the art that aspects of thepresent disclosure may be illustrated and described herein in any of anumber of patentable classes or context including any new and usefulprocess, machine, manufacture, or composition of matter, or any new anduseful improvement thereof. Therefore, aspects of the present disclosuremay be implemented entirely in hardware or combining software andhardware implementation that may all generally be referred to herein asa “circuit,” “module,” “component,” or “system” (including firmware,resident software, micro-code, etc.). Further, aspects of the presentdisclosure may take the form of a computer program product embodied inone or more computer readable media having computer readable programcode embodied thereon.

Any combination of one or more computer readable media may be utilized.The computer readable media may be a computer readable signal medium ora computer readable storage medium. For example, a computer readablestorage medium may be, but not limited to, an electronic, magnetic,optical, electromagnetic, or semiconductor system, apparatus, or device,or any suitable combination of the foregoing. More specific examples ofthe computer readable storage medium would include, but are not limitedto: a portable computer diskette, a hard disk, a random access memory(“RAM”), a read-only memory (“ROM”), an erasable programmable read-onlymemory (“EPROM” or Flash memory), an appropriate optical fiber with arepeater, a portable compact disc read-only memory (“CD-ROM”), anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing. Thus, a computer readable storage mediummay be any tangible medium that can contain, or store a program for useby or in connection with an instruction execution system, apparatus, ordevice.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, markup languages, style sheets and JavaScriptlibraries, including but not limited to Windows Presentation Foundation(WPF), HTML/CSS, XAML, and JQuery.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, systems andcomputer program products according to embodiments of the disclosure. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a computer interconnected with dedicated andspecial purpose electronic devices, or other dedicated programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create an integrated machine to implement thefunctions described in the flowchart and the block diagrams.

These computer program instructions may also be stored in a computerreadable medium that when executed can direct a computer, otherprogrammable data processing apparatus, or other devices to function ina particular manner, such that the instructions when stored in thecomputer readable medium produce an article of manufacture includinginstructions which when executed, cause a computer to implement thefunction/act specified in the flowchart and/or block diagram block orblocks. The computer program instructions may also be loaded onto acomputer, other programmable instruction execution apparatus, or otherdevices to cause a series of operational steps to be performed on thecomputer, other programmable apparatuses or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Referring to FIG. 1, system 100 includes network 101, content server 102connected to network 101, and camera/projector unit 104 connected tonetwork 101. Content server 102 is further connected to content database103 for the storage of multimedia and other relevant data.

In a preferred embodiment, network 101 is a wide area network such asthe internet, but can include other wide area and intranet networks.

In a preferred embodiment, camera/projector unit 104 downloads contentstored in content database 103 from content server 102 and saved tolocal memory for later projection. In another embodiment,camera/projector unit 104 streams content stored in content database 103from content server 102 for projection.

Referring to FIG. 2A in one embodiment, camera/projector unit 200includes enclosure 201, computer/processor 202, projector 203 connectedto computer/processor 202, and camera 204 connected tocomputer/processor 202.

Referring to FIG. 2B in another embodiment, camera/projector unit 205positions mirror 206 at angle α adjacent to projector 203 to redirectthe field of view of prof ector 203.

In a preferred embodiment, computer/processor 202 is a laptop computerhaving a suitable processor and memory for storage of content anddatabase, and any other relevant data. In another embodiment,computer/processor 202 is a tablet computer.

A projection application is stored in the memory of computer/processor202 for defining and projecting a set of trigger areas with projector203, detecting a user presence and recognizing the user with camera 204,detecting a selection of the set of trigger areas with camera 204, andprojecting content with projector 203, as will be further describedbelow.

In a preferred embodiment, camera 204 is a 3-D range camera, such as theKinect camera from Microsoft Corporation. Camera 204 provides image datain the form of streaming video at the rate of 30 frames per second in astandard 640×480 VGA. The focal length of the camera is 525 pixels witha field of view of approximately 63°. Camera 204 also providesdistance-to-target information from an on-board infraredtransmitter/receiver. The focal length of the infraredtransmitter/receiver is 580 pixels with a field of view of approximately58°. The infrared data is provided as a 16 bit number at a refresh rateof 200 μs.

Referring to FIG. 3, display 300 includes camera/projector unit 301having field of view 302. Camera/projector unit 301 is connected to arm304. Arm 304 is attached to and extends from an overhead ceiling. In apreferred embodiment, field of view 302 can range from approximately 45°to 75°. Product 303 is positioned in field of view 302.

Referring to FIG. 4 in another embodiment, display 400 includescamera/projector unit 401 having field of view 403. Camera/projectorunit 401 is connected to and supported by stand 402. Stand 402 isremovably connected to the floor. Product 404 is positioned in field ofview 403.

Referring to FIG. 5, display 500 includes camera/projector unit 501attached to arm 502 which is attached to base 508. Camera/projector unit501 projects trigger areas 504 and 505 onto the surface of product 503.When a user touches the surface of product 503 in trigger areas 504 and505, camera/projector unit 501 detects the presence of the user intrigger areas 504 and 505 and projects contents 506 and 507,respectively, onto the surface of product 503. In a preferredembodiment, each of contents 506 and 507 is a video. Any type ofgraphical images may be employed. For example, if a user touches thehood of the car shown in the drawing, an image of a running engine takenfrom the same perspective as the user may be projected on the hood tosimulate a “see through” effect to alert the user to “see” the runningengine through the hood.

Referring to FIG. 6 in another embodiment, display 600 includescamera/projector unit 601 having field of view 602 covering product 603.Camera/projector unit 601 projects trigger areas 604, 605, 606, and 607onto the surface of product 603. When a user touches the surface ofproduct in trigger areas 604, 605, 606, and 607, camera/projector unit601 projects content 615 onto the surface of product 603.

In one embodiment, if a user touches the surface of product 603 in anyof the trigger areas 604, 605, 606, and 607, camera/projector unit 601projects a different color/pattern 616 onto product 603, therebyenabling the user to visualize a potential change to the color orpattern of product 603.

Stand 608 is positioned adjacent to product 603 and is covered by fieldof view 602. Stand 608 includes placard 609, support 610 attached toplacard 609, and base 611 attached to support 610. Base 611 is removablyattached to the floor. Camera/projector unit 601 projects trigger areas612 and 613 onto the surface of placard 609. When a user touches thesurface of placard 609 in trigger areas 612 and 613, camera/projectorunit 601 projects content 614 onto placard 609, thereby transformingplacard 609 into an interactive digital sign. Content 614 preferablyincludes information, videos, and/or pictures of product 603. Content614 is retrieved from the content server or from the local memory ofcamera/projector unit 601.

Referring to FIG. 7, display 700 includes camera/projector units 701 and702. Camera/projector unit 701 has field of view 704 covering product703 including product side 706. Camera/projector unit 702 has field ofview 705 covering product 703 including product side 707.Camera/projector unit 701 projects trigger area 708 onto the surface ofproduct side 706. When a user touches the surface of product side 706 intrigger area 708, camera/projector unit 701 projects a video or graphicscontent onto the surface of product side 706. Camera/projector unit 702projects trigger area 709 onto the surface of product side 707. When theuser touches the surface of product side 707 in trigger area 709,camera/proj ector unit 702 projects a video or graphics content onto thesurface of product side 707.

Referring to FIG. 8 in another embodiment, display 800 includes product802 positioned in view of camera/projector unit 801. Camera/projectorunit 801 detects the position and presence of user 803 on product 802.In one embodiment, an identity of user 803 is determined through afacial recognition process, as will be further described below. Triggerareas 805 and 806 are projected by camera/projector 801 onto the surfaceof product 802 adjacent to user 803 in response to the presence of user803.

In one embodiment, content 804 is automatically projected onto thesurface of product 802 in response to the presence and identity of user803.

In another embodiment, content 804 is projected onto the surface ofproduct 802 in response to user 803 touching any of trigger areas 805and 806.

In another embodiment, camera/projector unit 801 projects a first videoas content 804 onto the surface of product 802 based on the detectedpresence of user 803. When user 803 touches any of trigger areas 805 and806, content 804 changes to a second video and a third video,respectively. For example, when camera/projector unit 801 detects thepresence of user 803 on product 802, content 804 is an introduction orwelcome video. When user 803 touches trigger area 805, content 804 isswitched to a graphical brochure. When user 803 touches trigger area806, content 804 is switched to a product animation. Any type of contentmay be employed for content 804.

Referring to FIG. 9 in another embodiment, display 900 includescamera/projector 901 and product 902. In this embodiment, projectedcontent 903 includes and sequentially displays the components of product902 including base layer 904, support layer 905, padding layer 906 andtop layer 907. In a preferred embodiment, projected content 903 includesa set of animations which represent a “pealing away” of each layer. Forexample, in response to a user interacting with a trigger area, toplayer 907 is “pealed away” to reveal padding layer 906. Each layer insuccession can be projected individually on product 902 thus enablingthe user to visualize the different layers of the product. In oneembodiment, projected content 903 covers the entirety of product 902. Inanother embodiment, projected content 903 covers a portion of product902.

Referring to FIG. 10 in another embodiment, display 1000 includescamera/projector 1001 and product 1002. Product 1002 rests on floor1004. Content 1003 is projected by camera/projector unit 1001 onto floor1004 surrounding product 1002. Content 1003 includes animation whichgives the appearance that product 1002 is driving through variousenvironments, such as rain, snow, or mud to give the appearance thatproduct 1002 is moving.

Referring to FIG. 11, method 1100 for calibrating camera/projector unit200 will be described. At step 1101, a baseline image of a surface isdetermined. In this step, the infrared transmitter/receiver of camera204 captures a baseline image including depth and color data of thesurface of a product. In one embodiment, a thin cover, such as a sheetmay be employed to cover a highly reflective surface, such as a surfaceof a car in order to capture an accurate baseline image.

At step 1102, a position for each of a set of trigger areas for theproduct is selected. In a preferred embodiment, each trigger area is anellipse. Other shapes may be employed. At step 1103, the set of triggerareas is projected on the surface. At step 1104, each trigger area iscalibrated by placing a three-dimensional object on the trigger area,such as a hemispherically shaped object. Other three-dimensional objectsmay be employed. In this step, a position of the three-dimensionalobject relative to the saved baseline image is captured withcamera/projector unit 200.

At step 1105, the captured position of each three-dimensional object issaved in the memory of camera/projector unit 200. At step 1106, apolygon is warped to match the surface, thereby ensuring that thepolygon conforms to the surface on which it will be projected. Thepolygon may correspond to the entire surface or a section of thesurface.

In a preferred embodiment, a non-affine transform is employed to map aset of points of the polygon to the surface. For example, if the polygonis a square in a flat plane and the surface is an angled surface withrespect to the projector, then the points of the square are mapped to aquadrilateral that distorts the square image so that when proj ected onthe angled surface, the projected quadrilateral appears as a square onthe angled surface. Other transforms, mapping, and distortion techniquesmay be employed.

At step 1107, the warped polygon is associated with the trigger area. Atstep 1108, a set of content is determined for the polygon. In this step,a file location of the desired set of content is associated with thepolygon. The set of content is downloaded and saved to local memory orlinked to a stream for the set of content. At step 1109, the set ofcontent is warped to match the warped polygon so that the selection ofthe trigger area projects the desired content undistorted on thesurface.

Referring to FIG. 12 in one embodiment, runtime process 1200 will bedescribed. At step 1201, runtime process 1200 begins. At step 1202, aset of trigger areas is projected onto a surface. At step 1203, a touchevent is determined, i.e., a determination is made whether a user is inany of the set of trigger areas. In this step, the camera is queried fora user position. The user position is compared to the coordinates of aboundary for the trigger area. If the user position is not within theboundary of the trigger area, then runtime process returns to step 1202.If the user position is within the boundary of the trigger area, thenruntime process proceeds to step 1204.

At step 1204, a polygon with a set of content associated with thetrigger area is projected onto the surface. At step 1205, the presenceof an end event is determined. An end event is a predetermined timeperiod or another touch event by the user in the set of trigger areas.The predetermined time period is the runtime of the set of content orany desired time period. If an end event has not occurred, then runtimeprocess 1200 returns to step 1204 to continue projecting the polygon andthe set of content. If an end event has occurred, then runtime process1200 proceeds to step 1206.

At step 1206, whether a terminate command has been received isdetermined. If a terminate command has not been received, then runtimeprocess 1200 returns to step 1202. If a terminate command has beenreceived, then runtime process 1200 ends at step 1207.

Referring to FIG. 13 in another embodiment, runtime process 1300 forprojection mapping will be described. At step 1301, a scan of an area,such as a room is performed by a camera to detect any surfaces. At step1302, the scan of a detected surface is mapped to a virtual area. Atstep 1303, a polygon and a set of content is warped to match the virtualarea.

In a preferred embodiment, a non-affine transform is employed to map aset of points of the polygon and the set of content to the surface. Inother embodiments, other transforms, mapping, and distortion techniquesmay be employed.

At step 1304, the camera is queried to determine a presence of a user inthe area. At step 1305, if a presence is not detected, then runtimeprocess 1300 returns to step 1304. If a presence is detected, then a setof trigger areas are projected onto the surface at step 1306. At step1307, a facial recognition process begins to recognize an identity ofthe user, as will be further described below.

At step 1308, if the identity of the user is recognized then runtimeprocess 1300 proceeds to step 1309. At step 1309, the polygon and a setof content associated with the user is projected onto the surface. Atstep 1311, the presence of an end event is determined. An end event is apredetermined time period or another touch event by the user in the setof trigger areas. The predetermined time period is the runtime of thecontent or any desired time period. If an end event has not occurred,then runtime process 1300 returns to step 1309 to continue projectingthe polygon and the set of content. If an end event has occurred, thenruntime process 1300 proceeds to step 1313.

If the identity of the user is not recognized in step 1308, then runtimeprocess 1300 proceeds to step 1310. At step 1310, the polygon and a setof predetermined content is projected onto the surface.

At step 1312, the presence of an end event is determined. If an endevent has not occurred, then runtime process 1300 returns to step 1310to continue projecting the polygon and the set of predetermined content.If an end event has occurred, then runtime process 1300 proceeds to step1313.

At step 1313, a touch event determination is made as to whether a useris in the set of trigger areas. In this step, the camera is queried fora user position. The user position is compared to the coordinates of aboundary for any of the set of trigger areas. If the position is notwithin the boundary of any of the set of the trigger areas, then runtimeprocess 1300 returns proceeds to step 1308. If the position is withinthe boundary of any of the set of trigger areas, then runtime processproceeds to step 1314.

At step 1314, a polygon with a set of content associated with thetrigger area is projected onto the surface. At step 1315, the presenceof an end event is determined. If an end event has not occurred, thenruntime process 1300 returns to step 1314 to continue projecting thepolygon and the set of content. If an end event has occurred, thenruntime process 1300 proceeds to step 1316.

At step 1316, whether a terminate command has been received isdetermined. If a terminate command has not been received, then runtimeprocess 1300 returns to step 1304. If a terminate command has beenreceived, then runtime process 1300 ends at step 1317.

Referring to FIG. 14, step 1307 will be further described as facialrecognition process 1400. Facial recognition process begins at step1401. At step 1402, a camera is queried to record the locations offacial features including eye location and color, ear location, chinlocation and hair color. In a preferred embodiment, the query isaccommodated by Microsoft SKD Rel. 7.1 .LIB fileMicrosoft.Kinect.Toolkit.FaceTracking. Any type of facial tracking maybe employed. In a preferred embodiment, at least 121 different locationsare determined. At step 1403, the distances between facial features aremeasured. At step 1404, a “facial map” is created. The facial mapincludes a table of the facial features and the distances between them.At step 1405, the facial map is compared to a set of digital imagesstored in memory. A result of the comparison is output at step 1406. Atstep 1407, the process ends.

Referring to FIG. 15, in one embodiment, components of interactiveprojection system 1500 are local computer 1502, camera 1504, andprojector 1506. Local computer 1502, camera 1504, and projector 1506 canbe separate components positioned individually or all be contained in asingle housing. Camera 1504 collects color and depth data within itsfield of view, constantly monitoring defined trigger areas forinteraction. Projector 1506 displays projectable content and triggerareas on any surface within its field of view. Local computer 1502stores and manipulates camera data, content files, locations ofprojected content, and locations of trigger areas and projected triggerareas. Local computer 1502 analyzes camera data to determine ifinteraction with any trigger areas has occurred as will be furtherdescribed. Local computer 1502 includes a transform function capable ofwarping content images in order to project the content onthree-dimensional surfaces undistorted as will be further described.

In a preferred embodiment, local computer 1502 contains a processor; amemory for storage of content, a database, and any other relevant data;and a software module including hardware drivers. Minimum requirementsfor local computer 1502 include an Intel Core 17 processor, at leasteight (8) GB of RAM, at least five (5) GB of local memory, a 2 GBgraphics processor, Windows operating system 8.1 or higher, and a USB3connection to camera 1504. An example of a suitable computer is theLatitude E6540 laptop computer available from Dell. In a preferredembodiment, hardware drivers include an image capturing program such asKinect for Windows available from Microsoft.

Local computer 1502 is connected to camera 1504. Camera 1504 is a 3-Drange camera. Camera 1504 includes firmware for movement interpretationand image recognition. Camera 1504 provides image color data in the formof streaming video at the rate of 30 frames per second in a standard640×480 VGA format at 4 bytes/pixel. The focal length of camera 1504 is525 pixels with a field of view that can range between approximately 45°to 75°. Camera 1504 also provides distance to target information from anon-board infrared transmitter/receiver pair. The focal length of theinfrared transmitter/receiver is 580 pixels with a field of view in therange of approximately 45° to 75°. The infrared data is provided as a 16bit number at a refresh rate of 200 μs. An example of a suitable camerais the Kinect camera available from Microsoft.

Local computer 1502 is connected to projector 1506 through an HDMI portand cable. Projector 1506 is a PC ready projector having a brightness of10,000 lumens and a resolution of 1920×1200 WUXGA. Projector 1506 has athrow distance in the range of 4 feet to 40 feet with a standard lens.Alternate lenses are available which can dramatically increase the throwdistance of projector 1506. Projector 1506 has a field of view angle inthe range of 45° to 75° but can be adjusted with different lenses. Anexample of a suitable projector is the PLWU8100F available fromViviBright.

Local computer 1502 is connected to microphone 1512 and speaker 1514.Microphone 1512 allow for the reception of audible commands that can beused to trigger the display or playback of images, video, and audio withrespect to one or more content areas. Speaker 1514 allows for theplayback of the audio channels associated with a video and for theplayback of audio files that have been associated with a trigger area.

In an alternate embodiment, components of interactive projection system1500 further include network 1510, content server 1522, database 1524,and control computer 1520. Content server 1522 is connected to network1510. Network 1510 connects local computer 1502 to content server 1522.Content server 1522 is further connected to database 1524 for thestorage of multimedia and other relevant data. Network 1510 is a widearea network such as the Internet but can include other wide area orintranet networks. Local computer 1502 can download content stored indatabase 1524 from content server 1522 and save the content to localmemory for later projection. Local computer 1502 can also stream contentstored in database 1524 from content server 1522 for projection withprojector 1506.

Control computer 1520 is connected to content server 1522 and connectedto local computer 1502 through network 1510. Through control computer1520, a user can manipulate local computer 1502 and set up interactiveprojection system 1500 from a remote location. Minimum requirements forcontrol computer 1520 include an Intel Core 17 processor, at least eight(8) GB of RAM, at least five (5) GB of local memory, a 2 GB graphicsprocessor, and Windows operating system 8.1 or higher.

Referring to FIG. 16A and coordinate system 1630, the X axis representshorizontal, the Y axis represents vertical, and the Z axis is out of thepage. The relative positions of the components within a viewing space1602 are shown. Viewing space 1602 is a room having a light levelsufficient for the projection of projector 1506 to be adequately viewedby a user. The dimensions of viewing space 1602 are those of a typicalshowroom floor but can vary greatly dependent on the size of the objectof the interactive display and the lens of the projector. In general,camera 1504 is positioned to achieve field of view 1604. Field of view1604 can range from 45°-75°. Camera 1504 is suspended from ceiling 1640with arm 1606 but can also be attached to a floor stand with equalsuccess. Object 1608 is positioned on floor 1610 within field of view1604. The locations of any trigger areas must be within field of view1604.

In general, projector 1506 is positioned to achieve field of view 1624.Field of view 1624 can range from 45°-75°. Projector 1506 may besuspended from ceiling 1640 with arm 1616 but can also extend from afloor stand. Object 1608 is positioned on floor 1610 within field ofview 1624. The location of any projected content must be within field ofview 1604. Projector 1506 and camera 1504 are separated by a distance1662 along the X axis and by distance 1664 along the Y axis.

Placard 1622 extends from floor 1610 via stand 1628. As shown, contentmay be displayed on placard 1622 because placard 1622 is positionedwithin field of view 1624 of projector 1506. In another embodiment,placard 1622 is positioned within field of view 1604 of camera 1504. Inanother embodiment, placard 1622 is in field of view 1624 of projector1506 and field of view 1604 of camera 1504 and can display both contentareas and trigger areas. In a preferred embodiment, the total height1636 of the stand and placard is about 3 to 4 feet, the length 1638 ofthe placard is about 16 inches, and the width 1652 of the placard isabout 12 inches. Distance 1650 is the distance between ceiling 1640 andfloor 1610 and is about 16 feet in one preferred embodiment.

Field of view 1624 of projector 1506 is about 72 degrees and includesfield of view 1642 and field of view 1644. Field of view 1642 is thefield of view for content area 1632 on the surface of object 1608 and isabout 7 degrees. Field of view 1644 is the field of view for contentarea 1634 on placard 1622 and is about 5 degrees. Placard 1622 andobject 1608 are within distance 1660 of about 8 feet of each other sothat the images projected onto placard 1622 and onto the object arewithin field of view 1624 of projector 1506.

Field of view 1604 of camera 1504 includes field of view 1646 and fieldof view 1648. Field of view 1646 is the field of view for trigger area1620 on the surface of object 1608 and is about 4 degrees. Field of view1648 is the field of view for trigger area 1626 on the surface of object1608 and is about 3 degrees. Field of view 1604 of camera 1504 isdifferent from field of view 1624 of projector 1506 so that camera 1504may monitor trigger areas that are outside of field of view 1624 ofprojector 1506.

Referring to FIG. 16B, camera 1504 is positioned generally over object1608. The position of camera 1504 on the X axis and Z axis is dictatedby the desired positions of trigger areas. In one embodiment, camera1504 is positioned within a range of 5 to 16 feet along the Y axis aboveobject 1608 and within a range of 10 to 20 feet of floor 1610. Triggerareas shown within the field of view of the camera. Trigger area 1620 ispositioned on object 1608. Trigger area 1626 is positioned on thesurface of object 1608. The system may project an indicator of triggerarea 1626 on the surface of object 1608

The position of projector 1506 on the X and Z axes is dictated by thedesired positions of content areas 1632 and 1634. In one embodiment,projector 1506 is positioned within a range of 4 to 40 feet along the Yaxis above any surface in which content is desired to be displayed. Theideal distance being dictated by the lens. The surface on which thesystem displays content should be within the field of view of theprojector. In one embodiment, content area 1632 is positioned on thehood portion of object 1608, which in this case is a car. Content area1634 is positioned on placard 1622.

Flat surface 1654, shown in FIG. 16C, is a hypothetical surface to betargeted at object 1608. Object 1608 includes curved surface 1656, ontowhich content area 1632 is projected. Angle 1658 is the differencebetween flat surface 1654 and curved surface 1656.

The trigger areas are distinct from the content areas but may be linkedwith each other. In an alternative embodiment where a trigger area is onthe headlight of a car, the associated content area may be on the hoodof the car so that when the trigger area on the headlight is touched, avideo is played on the content area on the hood and the audio associatedwith the video is played via one or more speakers. Furthermore, insteadof a video being played on the hood, the trigger area on the headlightmay be linked to an audio recording so that when the trigger area on theheadlight is touched, the audio recording (also known as an audioresponse) is played.

In additional embodiments, a computer monitoring the trigger areas isalso in communication with the car and can control the functions of thecar in response to interaction with the trigger areas. This allows forcontrol of the object—the car—in concert with media displayed orprojected onto the object. In one embodiment, interaction with a triggerarea on the hood of a car causes a video of the engine to be played onthe hood and causes the system to start up the engine of the car. Whenthe object is a car, additional features can be controlled in responseto interaction with trigger areas, such as: turning on or off headlightsand taillights, locking and unlocking doors, opening and closingwindows, opening the trunk, and the like.

Referring to FIG. 16C, surface 1654 is a hypothetical flat surface aboveobject 1608 through which an image from the projected passes. The planeof surface 1654 is perpendicular to the direction that the projector ispointing and is tangential to object 1608.

Referring to FIG. 16D, surface 1656 is the surface of object 1608, whichis curved and not flat. Images projected onto curved surfaces, such assurface 1656, instead of onto flat surfaces, such as hypotheticalsurface 1654, are adjusted by warping and transforming the image tomatch the curvature of the surface.

Referring to FIG. 17A, trigger area is described. Trigger areas 1702 and1704 are predefined 3-dimensional spaces stored in the database. Triggerareas are monitored by the infrared sensors in camera 1504 forinteraction with the system by a user. When the infrared sensors detecta user in a trigger area, a predefined action, such as displaying storedcontent, is carried out by the system.

A trigger area can be any three-dimensional shape bounded by a series ofpoints which form lines that enclose a volume. Referring to coordinatesystem 1706, trigger area 1702 is bounded by a set of points includingX₁, Y₁, Z₁; X₂, Y₁, Z₁; X₁, Y₂, Z₁; X₂, Y₂, Z₁; and function 1708.Function 1708, in this example, is a hyperbolic function defined by theequation

C≈x²+y²   Eq. 12

Other hyperbolic functions or linear functions may be used to define anyset of points, in the X, Y, or Z directions, defining a general shape,so long as the function is closed with respect to the discrete area.Trigger area 1704 is a three-dimensional polygon. The various X, Y, andZ values for each trigger area can vary. The X and Y values rangebetween 2 and 12 inches, preferably around 7 to 8 inches. The Z valuecan range between 1 and 4 inches, preferably about 2 inches.

FIG. 18A is a diagram of a set of data elements stored in the computerand accessed by a trigger area software program, such as touchable areaapplication 1802, in accordance with one embodiment. Touchable areaapplication 1802 is an application running on a computer system thatallows for the display of projected content onto objects and surfacesbased user interaction with trigger areas that have been associated withand linked to the projected content. Touchable area application 1802includes tabs 1804, 1806, and 1808; one or more content controls 1852;and one or more trigger controls 1854.

Tabs 1804, 1806, and 1808 are user interface elements displayed in theapplication window of touchable area application 1802 that allow for theselection between different modes of the application, such as modes1910, 1912, and 1914 shown in FIG. 19A. Tabs 1804, 1806, and 1808 aredisplayed in display area 1816.

Display area 1816 is the display space of touchable area application1802. Touchable Area Application includes trigger controls 1854 andcontent controls 1852 that are displayed in display area 1816.

Trigger controls 1854 are each associated with an instance of triggerarea data structure 1818. Trigger area data structure 1818 includeseveral fields of data to identify a trigger area that is monitored by acamera with the computer, such as camera 1504 and computer 1502 of FIG.15.

Content controls 1852 are each associated with an instance of contentdata structure 1820. Content data structure 1820 identifies the contentand the content area in which the content is displayed by the system.Content data structure 1820 includes several data fields to identify thecontent and control its display.

FIG. 18B is a diagram of a trigger area data structure in accordancewith one embodiment of the disclosure. Trigger area data structure 1818is associated with trigger control 1854 and includes coordinates 1822,type 1824, name 1826, identifier 1828, depth data 1830, content datastructure 1820, and filename 1834. The data associated with trigger areadata structure 1818 is stored to one or more files that allows triggerdata structure area 1818 to be saved and reloaded by touchable areaapplication 1802.

Coordinates 1822 identify x and y locations for each of four corners forwhere trigger control 1852 is displayed in display area 1816.

Type 1824 identifies a type of content associated directly with triggerarea data structure 1818, such as an image or video.

Name 1826 is a user friendly name that identifies an instance of atrigger control.

Identifier 1828 is a unique identifier, such as a globally uniqueidentifier (GUID), which is given to each trigger control 1854 oftouchable area application 1802 to access trigger control 1854.

Depth data 1830 identifies the three-dimensional location or touch boxassociated with trigger area data structure 1818 and trigger control1854.

Content data structure 1820 identifies the content area that isassociated with trigger area data structure 1818 and trigger control1854.

Filename 1834 identifies a file, such as an image file, that provides adefault image for trigger control 1854. Filename 1834 may point to afully transparent image so that, by default, no image is displayed bytrigger control 1854.

FIG. 18C is a diagram of a content data structure in accordance with onepreferred embodiment. Content data structure 1820 is associated withcontent control 1852 and includes several data fields to identify anddisplay content within display area 1816 of touchable area application1802.

Coordinates 1836 identify x and y locations for each of the corners ofcontent control 1852. The corners identified by coordinates 1836 form apolygon. When the image or video associated with content data structure1820 is displayed, the corners of the image or video are warped andtransformed to fit the shape identified by coordinates 1836.

Type 1838 identifies the type of content associated with content datastructure 1820 and content control 1852. Content types include image,video, sound, and the like.

Loop 1840 identifies how many times to loop the content when, forexample, type 1838 of content area 1820 identifies that the content is avideo.

Orientation 1842 identifies the orientation used to display imagesassociated with content control 1852. In one embodiment, Orientation1842 specifies that, when displayed, the image or video associated withcontent control 1852 is flipped about the horizontal or vertical axes orrotated by a multiple of 90 degrees.

Filename 1844 identifies the filename of the image or video associatedwith content data structure 1820 and content control 1852. In additionalembodiments, filename 1844 is a link to static or dynamic content suchas a video file, a streaming video file, or a live camera stream that isplayed in response to interaction with a trigger area.

Identifier 1846 is a unique identifier or GUID that is given to eachcontent control 1852 of touchable area application 1802 to accesscontent control 1852.

Name 1848 is a user friendly name that identifies an instance of acontent control.

FIG. 19A is a data flow diagram of a system calibrating trigger areasand content areas in accordance with the disclosure. In this embodiment,the system includes computer 1902, camera 1904, projector 1906, anddisplay 1908.

Computer 1902 is connected to range camera 1904 via a USB (universalserial bus) 3.0 connection and is connected to projector 1906 anddisplay 1908 by an HDMI (high-definition multimedia interface)connection. Computer 1902 controls the images displayed by display 1908and projector 1906. Camera 1904 provides color data and depth data.Projector 1906 projects images sent from computer 1902 onto any surfacein front of projector 1906, including one or more display stands,objects, and screens. Display 1908 displays the application to aid thecalibration process.

The touchable area application uses three modes to perform thecalibration process, first mode 1910, second mode 1912, and third mode1914. When first mode 1910 is selected, user interface controls areassociated with the touchable area application. For example, in the casewhere a static image is displayed by default and two videos are shownbased on interaction with two trigger areas, the static image, the twovideos, and the two trigger controls are added to the touchable areaapplication. In First mode 1910, the application performs steps 1916,1918, 1920, and 1922.

In second mode 1912, the three-dimensional real world locations fortrigger areas are selected. In second mode 1912, the applicationperforms steps 1924, 1926, 1928, 1930, and 1932.

In third mode 1914, the projected locations of the projected content isselected, fine-tuned, and aligned. In third mode 1914 the applicationperforms steps 1934, 1936, 1938, and 1940.

At step 1916, a first tab is selected in the touchable area applicationto set the mode of the touchable area application to first mode 1910.One example of code for selecting a first tab is shown in the computerprogram listing appendix in the computer program listing of the filenamed MainWindow.xaml.cs at lines 922-926.

At step 1918, the display image of the application is sent to display1908 so that the user interface controls—trigger controls and contentcontrols—can be added and manipulated.

At step 1920, trigger controls and content controls are added andupdated. New trigger controls and content controls may be added and theassociations between the trigger controls, trigger areas, contentcontrols, and content areas may be updated. The corners of the contentcontrols and trigger controls can be moved or dragged to reshape theimages and video that are projected onto the content areas and triggerareas. One example of code that performs this function is shown in thecomputer program listing appendix in the computer program listing of thefile named NonAffineTransformControl.xaml.cs at lines 437-476. This codecalls a function to determine a transformation matrix, which is shown inthe computer program listing appendix in the computer program listing ofthe file named NonAffineTransformControl.xaml.cs at lines 1220-1258. Oneexample of code for updating trigger or content controls is shown in thecomputer program listing appendix in the computer program listing of thefile named NonAffineTransformControl.xaml.cs at lines 478-512.

At step 1922, for each change to the user interface controls from step1920, the image sent to display 1908 is updated. This provides feedbackto allow the sequences, timing, and interaction between the triggercontrols and content controls.

At step 1924, a second tab is selected to shift the application tosecond mode 1912 to allow for the selection of the physical locations ofthe trigger areas that are associated with trigger controls. One exampleof code for selecting a second tab is shown in the computer programlisting appendix in the computer program listing of the file namedMainWindow.xaml.cs at lines 927-971.

At step 1926, computer 1902 receives color and depth data from camera1904.

At step 1928, the display area of the application window of theapplication is changed to show a combination of the color and depth datareceived from camera 1904 with the trigger controls overlaid onto theimage.

At step 1930, the trigger controls that were added in first mode 1910are associated physical locations that are monitored by camera 1904. Theassociations are created by dragging and moving the trigger controlsdisplayed on display 1908. One example of code for updating triggercontrols is shown in the computer program listing appendix in thecomputer program listing of the file namedNonAffineTransformControl.xaml.cs at lines 478-512.

At step 1932, for each change to a trigger control and each change inthe color and depth data received from camera 1904, computer 1902updates the image sent to display 1908.

At step 1934, a third tab, such as third tab 1808 of FIG. 18, isselected to set the mode of the application to third mode 1914 to allowfor selection of the location of the content areas that will beassociated with the content controls. One example of code for selectinga third tab is shown in the computer program listing appendix in thecomputer program listing of the file named MainWindow.xaml.cs at lines916-920.

At step 1936, computer 1902 stops displaying the color and depth datafrom camera 1904 and displays the touchable area application as a fullscreen window on projector 1906 including the user interface controls.Projector 1906 projects the image from computer 1902 onto the surfacesand objects in front of projector 1906. For example, when camera 1904projects onto the hood of a car the content displayed by the touchablearea application is an image of the engine. In an alternativeembodiment, the image sent by computer 1902 to projector 1906 is alsosent to display 1908 so that the image is displayed by both display 1908and by projector 1906.

At step 1938, the content control is updated. For example, the sizes ofthe content control are adjusted to match the surfaces onto which theyare being projected. When the hood of the car has a trapezoidal shape,the corners of the content control are dragged to match the trapezoidalshape of the hood. One example of code for updating content controls isshown in the computer program listing appendix in the computer programlisting of the file named NonAffineTransformControl.xaml.cs at lines478-512.

At step 1940, for each update to the content controls from step 1938,the image sent to the projector is updated to include the changes madein step 1938.

FIG. 19B is a flow chart for updating a trigger area associated with atrigger control in accordance with one embodiment of the disclosure. Themethod for updating a trigger area of FIG. 19B is one embodiment forstep 1930 of FIG. 19A.

At step 1942, color and depth data are received from a camera, such ascamera 1904. The color and depth data are received as one or more bytestreams that contain two frames of data, one based on the color sensorof the camera and one based on the infrared depth sensor of the camera.The resolution of the color sensor and the depth sensor differ so thatthe resolution of the two frames of data also differ. As one example,the color data may have a resolution of 1920 by 1080 with 32 bits ofcolor information per pixel and the depth data may have a resolution of640 by 480 with 16 bits of depth information per pixel.

At step 1944, a combination of the color and depth data is displayed.The color data resolution is cropped to the depth data resolution.Within the cropped resolution, color data is displayed if the depth dataindicates there is a surface or object within the range of the depthsensor.

At step 1946, a trigger area associated with a trigger control isselected. The trigger area identifies the three-dimensional coordinatesof a trigger area that is stored in a trigger area data structureassociated with a trigger control and is with respect to the camera thatprovides the depth data. As shown in FIG. 19C, area 1952 is an area of asurface within the range and viewing area of the depth sensor. In apreferred embodiment, the shape of the area defaults to a square orrectangle, but any other shape can be used.

At step 1948, a depth for the trigger area is selected. The depthcombined with the trigger area creates a volume that is associated witha trigger area and is used to determine if the trigger area has beentouched. FIG. 19D shows volume 1954 created from area 1952 through theaddition of depth 1956.

At step 1950, the trigger area and depth are stored. The trigger areaand depth are associated with the trigger area and saved to thedatabase. In a preferred embodiment, the trigger area and depth arestored in depth data 1830 of trigger area data structure 1818 of FIG.18B.

FIG. 19E is a flow chart for updating a content area in accordance withone embodiment of the disclosure. The method for updating a content areaof FIG. 19E is one embodiment for step 1938 of FIG. 19A.

At step 1958, an initial flat projection area is selected. The nativeresolution of the content that is selected and associated with a contentcontrol provides the initial flat projection area. FIG. 19F showsinitial flat projection 1968 of content that has been selected.

At step 1960, an area related to an affine transformation is selected.In a preferred embodiment, the area for the affine transformation isselected by dragging the corners the content area into a desired shape.As an example, when the content area is a rectangular image, one or moreof the corners are dragged to form a scaled parallelogram shape, asshown by transformation 1970 of FIG. 19F.

At step 1962, an area related to a non-affine transformation isselected. In a preferred embodiment, the area is selected by draggingone or more of the corners and lines that from the boundary of thecontent area into curved shapes, as shown by transformation 1972 of FIG.19F.

At step 1964, a brush or filter is created from the affinetransformation and the non-affine transformation. The brush will beapplied to the content associated with the content control to transformthe original size and shape of the content to the selected size andshape of the area selected via steps 1960 and 1962 that correspond to anaffine transformation and a non-affine transformation. When the contentis a video, the brush is applied to each frame of the video as the videois displayed.

At step 1966, the brush created from the selected areas for the affinetransformation and the non-affine transformation are saved to persistentstorage

FIG. 19F shows the matrix transformations applied to content inaccordance with one embodiment of the disclosure. Flat projection 1968is an initial projection that has not been warped and transformed.Projection 1970 has been warped and transformed using an affinetransformation. Projection 1972 has been warped and transformed using anon-affine transformation.

The following code listing provides an example of calculating atransformation:

// The input array of points describes a 2D rectangle // (with Z assumedto be zero) in the order // lower-left, upper-left, lower-right,upper-right. // The returned transform maps the points (0, 0, 0), // (0,1, 0), (1, 0, 0), and (1, 1, 0) to these points. private Matrix3DCalculateNonAffineTransform(Point3D[ ] points) { // Affine transform //---------------- // This matrix maps (0, 0) --> (x0, y0) //    (0, 1)--> (x1, y1) //    (1, 0) --> (x2, y2) //    (1, 1) --> (x2 + x1 + x0,y2 + y1 + y0) Matrix3D A = new Matrix3D( ); A.M11 = points[2].X −points[0].X; A.M12 = points[2].Y − points[0].Y; A.M21 = points[1].X −points[0].X; A.M22 = points[1].Y − points[0].Y; A.OffsetX = points[0].X;A.OffsetY = points[0].Y; // Calculate point (a, b) that get mapped bythe affine // transform to (x3, y3) double den = A.M11 * A.M22 − A.M12 *A.M21; double a = (A.M22 * points[3].X − A.M21 * points[3].Y +     A.M21 * A.OffsetY − A.M22 * A.OffsetX) / den; double b = (A.M11 *points[3].Y − A.M12 * points[3].X +      A.M12 * A. OffsetX − A.M11 *A.OffsetY) / den; // Non-affine transform // -------------------- //This matrix maps (0, 0) --> (0, 0) //    (0, 1) --> (0, 1) //    (1, 0)--> (1, 0) //    (1, 1) --> (a, b) Matrix3D B = new Matrix3D( ); B.M11 =a / (a + b − 1); B.M22 = b / (a + b − 1); B.M14 = B.M11 − 1; B.M24 =B.M22 − 1; return B * A; }

FIG. 20A is a data flow diagram of a system using trigger areas inaccordance with the disclosure. The system includes computer 2002, rangecamera 2004, and projector 2006 and has already been calibrated to alignthe trigger areas and content areas to the surfaces and objects in frontof projector 2006.

Computer 2002 is connected to range camera 2004 via a USB (universalserial bus) 3.0 connection and is connected to projector 2006 by an HDMI(high-definition multimedia interface) connection. Computer 2002controls the image displayed by projector 2006 based on data receivedfrom range camera 2004. Range camera 2004 provides color data and depthdata. Projector 2006 projects images sent from computer 2002 onto anysurface in front of projector 2006, including one or more displaystands, objects, and screens.

At step 2008, computer 2002 sends the display image that will bereceived and displayed by projector 2006. The display image has aresolution of, e.g., 1920 by 1080 pixels, which is a native resolutionfor projector 2006.

At step 2010, a frame of data is sent from range camera 2004 andreceived by computer 2002. The color data and the depth data aremeasured by range camera 2004 and are separated into two different bytestreams with different resolutions. The color data is 1920 by 1080 by 32bits with 8 bits for each of red, green, blue, and alpha channels. Thealpha channel defaults to 1. The depth data is 640 by 480 by 16 bitswith 13 bits for depth information that measures depth from range camera2004 in 1 millimeter increments.

At step 2012, computer 2002 determines if a trigger area has beentouched. The determination is made by comparing a base line of the depthdata received from range camera 2004 with the current depth datareceived from range camera 2004. In one preferred embodiment, thebaseline depth data is an average of three frames of data capturedduring the calibration process.

In one embodiment, detection of a trigger area is determined bycalculating whether 20% or more of a volume associated with the triggerarea is occupied for at least three frames in a row. One example of codefor performing this function is shown in the computer program listingappendix in the computer program listing of the file namedMainWindow.xaml.cs at lines 494-527. The occupation of the volume isdetermined by the depth associated with the pixels related to the volumeof the trigger area. The depth should be less than the baseline depth,but greater than a threshold depth so that a trigger area is onlyactivated by an object that is near the real world location of thetrigger area.

In another embodiment, for each pixel of a trigger area that isoccupied, the surrounding eight pixels (forming a 3 by 3 or nine pixelmatrix) are tested to see if they are each similarly occupied. Thisadditional precision allows for the touch of a single finger to activatea trigger area.

At step 2014, computer 2002 updates the display image to be sent toprojector 2006 based on frame data received from range camera 2004. Whenno trigger areas are clicked or activated, the display image is updatedfor content areas that includes video. When a trigger area is properlyactivated, the content areas associated with that trigger area areactivated.

At step 2016, computer 2002 sends the updated image. The updated imageis based on the content being displayed and the interaction with triggerareas.

FIG. 20B is a flow chart for determining a touch of a trigger area inaccordance with one embodiment of the disclosure. One example of codefor performing steps analogous to the steps of the flow chart of FIG.20B is shown in the computer program listing appendix in the computerprogram listing of the file named MainWindow.xaml.cs at lines 494-527.

At step 2018, depth data is received from a camera, such as range camera2004 of FIG. 20. In a preferred embodiment, the depth data has aresolution of 640 by 480 pixels with 13 bits of depth information perpixel and aligned to 16 bits.

At step 2020, a determination is made as to whether 20% or more of thetrigger volume associated with a trigger area has been filled by anobject. Each pixel of the depth data related to the trigger volume iscompared to its corresponding baseline depth to determine of the pixelincludes an object that is a threshold amount above the baseline depth.When 20% or more of the pixels associated with the trigger volume haveare determined to include an object, then a frame counter is incrementedand the method proceeds to step 2022. Otherwise, the frame counter isdecremented to a minimum value of zero and the method returns to step2018 to wait for the next frame of depth data.

At step 2022, the frame count for the trigger area is compared. In apreferred embodiment, an object must be detected in relation to thetouchable are for three consecutive frame of data. When it has beendetected for three consecutive frames of data, the method proceeds tostep 2024. Otherwise, the method returns to step 2018 to wait from thenext frame of depth data.

At step 2024, an object of sufficient size has been detected asufficient number of times to activate a touch event that is associatedwith the trigger area. In a preferred embodiment the touch event is amouse click event that is associated with a trigger control associatedwith the trigger area that causes the display of content via a contentcontrol onto the content area. The content displayed in the content areamay be static, dynamic, or streamed.

In FIGS. 21A and 21B, a system is being calibrated to display contentonto an object. A touchable area application, such as touchable areaapplication 1802 of FIG. 18, includes tabs 2116, 2118, and 2120 thatallow for switching between different modes—such as modes 1910, 1912,and 1914 of FIG. 19A—in the touchable area application for calibration.The first mode to add, associate, and manipulate trigger areas andcontent areas is accessed by selecting tab 2116. The second mode toidentify the three-dimensional real world location of the trigger areasassociated with trigger controls is accessed by selecting tab 2118. Thethird mode to display, align, and manipulate the content areas withrespect to the object is accessed by selecting tab 2120.

FIG. 21A is a view of the application when a first tab is selected. Whenfirst tab 2116 is selected, the application window 2102 is seen on afirst display, which is connected to and may be substantially co-locatedwith a computer, such as computer 1902 of FIG. 19A. Displayingapplication window 2102 on the first display allows a user to set up thetrigger areas and content areas for the system along with the linkingand interaction behaviors associated with the trigger controls, contentcontrols, trigger areas, and content areas. Trigger controls 2104 and2106 are displayed in application window 2102. Application window 2102also displays content control 2108, content control 2110, contentcontrol 2112 and mouse pointer 2114. Application window 2102 alsoincludes tabs 2116, 2118, and 2120. Background 2128 of applicationwindow 2102 is black.

Trigger control 2104 is associated with content control 2110. When thesystem is running and trigger control 2104 is touched, the content ofcontent control 2110 will be played on top of the content of contentcontrol 2108. Similarly, trigger control 2106 is associated with contentcontrol 2112 so that touching or clicking trigger control 2106 plays themedia or content of content control 2112 on top of the content ofcontent control 2108.

Content control 2108 includes the default content that is normallydisplayed by the system. In the embodiment of FIGS. 21A and 21B, contentcontrol 2108 is linked to a static image of an engine, such as theengine of car 2122. Content control 2110 is linked to trigger control2104 and includes a video of the performance characteristics of theobject, i.e., car 2122 of FIGS. 21B. Content control 2112 is linked to avideo of the safety systems of car 2122 and is associated with triggercontrol 2106.

FIG. 21B is a view of the application when second tab 2118 is selected.When second tab 2118 is selected, instead of showing content controls onbackground 2128, application window 2102 shows a combination of colorand depth data from the camera. When there is no depth data for a pixel,the pixel is shown as black. Otherwise, the RGB (red green blue) colorvalue from the camera is shown. In one embodiment when the resolution ofthe color data does not match the resolution of the depth data, parts ofthe image where there is no color data are shown only using the depthdata, such as in areas 2130 and 2132. Area 2134 is shown using thecombination of color and depth data and areas 2130 and 2132 are shownusing only depth data since there is no color data for that portion.Alternative embodiments show any one or combination of color data anddepth data on application window 2102.

Trigger controls 2104 and 2106 can be dragged to any part of the viewshown application window 2102 in FIG. 21B. When a trigger control ismoved, it is associated with the three-dimensional physical locationthat corresponds with the pixels displayed in application windows 2102in FIG. 21B. A touch box is created for the trigger area associated withthe trigger control so that when it is determined that an objectsufficiently fills the touch box, the trigger control is activated bycreating a virtual mouse click for the trigger control associated withthe trigger area in the display area of the touchable area application.

In an additional embodiment, when the projector can display an image inview of the camera, the projector flashes an image related to the sizeand location of the trigger area. When the flashing image related to thetrigger area is identified from the color data from the camera, thethree dimensional coordinates that correspond to the trigger area areidentified from the depth data and are associated with the triggercontrol to automatically calibrate the three dimensional location of thetrigger area with the trigger control.

It will be appreciated by those skilled in the art that changes could bemade to the embodiments described above without departing from the broadinventive concept. It is understood, therefore, that this disclosure isnot limited to the particular embodiments herein, but it is intended tocover modifications within the spirit and scope of the presentdisclosure as defined by the appended claims.

1-20. (canceled)
 21. A system for interactively updating a display on asurface: a computer; a camera; a projector; the computer configured toperform the steps of: sending a first image file to the projectorthereby causing the projector to display a first image on the surface;receiving a frame of data from the camera; retrieving a set of depthdata associated with the frame of data; determining if a trigger area inthe set of depth data has been activated; and, if the trigger area hasbeen activated, then sending a second image file, based on the frame ofdata, to the projector, thereby causing the projector to display asecond image on the surface.
 22. The system of claim 21 wherein the stepof sending a first image file further comprises: selecting a flatprojector area as a base image file; performing an affine transform onthe base image file to create an intermediate image file; and performinga non-affine transform on the intermediate image file to derive thefirst image file.
 23. The system of claim 22: wherein the step ofdetermining further comprises: comparing a pixel of the depth data thatis related to a trigger volume that is associated with the trigger areato a corresponding baseline depth to determine if the pixel includes anobject that is a threshold amount above the baseline depth.
 24. Thesystem of claim 23: wherein the step of determining further comprises:for a pixel of the trigger area that is occupied, testing eight pixelsthat surround the occupied pixel to confirm that each of the eightsurrounding pixels are also occupied.
 25. The system of claim 23:wherein the step of determining further comprises: calculating whether20% or more of the trigger volume associated with the trigger area isoccupied for at least three frames of data.
 26. The system of claim 23,further comprising: a display; the computer further configured toperform the steps of: receiving a selection of a first tab of anapplication; sending a control image to the display based on theselection of the first tab; adding a trigger control to the application;and, updating the trigger control based on a movement of a corner of thetrigger control.
 27. The system of claim 26: wherein the computer isfurther configured to perform the steps of: receiving a selection of asecond tab of the application; updating the control image sent to thedisplay to include the color data and the depth data after the secondtab is selected; and, associating the trigger area and the triggervolume with the trigger control.
 28. A method comprising updating adisplay on a surface: sending a first image file to a projector therebycausing the projector to display a first image on the surface; receivinga frame of data from a camera; retrieving a set of depth data associatedwith the frame of data; determining if a trigger area in the set ofdepth data has been activated; and, if the trigger area has beenactivated, then sending a second image file, based on the frame of data,to the projector, thereby causing the projector to display a secondimage on the surface.
 29. The method of claim 28 wherein the step ofsending a first image file further comprises: selecting a flat projectorarea as a base image file; performing an affine transform on the baseimage file to create an intermediate image file; and performing anon-affine transform on the intermediate image file to derive the firstimage file.
 30. The method of claim 29: wherein the step of determiningfurther comprises: comparing a pixel of the depth data that is relatedto a trigger volume that is associated with the trigger area to acorresponding baseline depth to determine if the pixel includes anobject that is a threshold amount above the baseline depth.
 31. Themethod of claim 30: wherein the step of determining further comprises:for a pixel of the trigger area that is occupied, testing eight pixelsthat surround the occupied pixel to confirm that each of the eightsurrounding pixels are also occupied.
 32. The method of claim 30:wherein the step of determining further comprises: calculating whetheror not 20% or more of the trigger volume associated with the triggerarea is occupied for at least three frames of data.
 33. The method ofclaim 30, further comprising: receiving a selection of a first tab of anapplication; sending a control image to a display based on the selectionof the first tab; adding a trigger control to the application; and,updating the trigger control based on a movement of a corner of thetrigger control.
 34. The method of claim 33, further comprising:receiving a selection of a second tab of the application; updating thecontrol image sent to the display to include the color data and thedepth data after the second tab is selected; and, associating thetrigger area and the trigger volume with the trigger control.
 35. Acomputer readable medium comprising computer program instructions thatwhen executed cause a computer to perform the steps of: sending a firstimage file to the projector thereby causing the projector to display afirst image on a surface; receiving a frame of data from a camera;retrieving a set of depth data associated with the frame of data;determining if a trigger area in the set of depth data has beenactivated; and, if the trigger area has been activated, then sending asecond image file, based on the frame of data, to the projector, therebycausing the projector to display a second image on the surface.
 36. Thecomputer readable medium of claim 35 wherein the step of sending a firstimage file further comprises: selecting a flat projector area as a baseimage file; performing an affine transform on the base image file tocreate an intermediate image file; and performing a non-affine transformon the intermediate image file to derive the first image file.
 37. Thecomputer readable medium of claim 36: wherein the step of determiningfurther comprises: comparing, by the computer, a pixel of the depth datathat is related to a trigger volume that is associated with the triggerarea to a corresponding baseline depth to determine if the pixelincludes an object that is a threshold amount above the baseline depth.38. The computer readable medium of claim 37: wherein the step ofdetermining if the trigger area has been activated further comprises:for a pixel of the trigger area that is occupied, testing, by thecomputer, eight pixels that surround the occupied pixel to confirm thateach of the eight surrounding pixels are also occupied.
 39. The computerreadable medium of claim 37: wherein the step of determining furthercomprises: calculating, by the computer, whether 20% or more of thetrigger volume associated with the trigger area is occupied for at leastthree frames of data.
 40. The computer readable medium of claim 37,further comprising: receiving, by the computer, a selection of a firsttab of an application; sending, by the computer, a control image to adisplay based on the selection of the first tab; adding, by thecomputer, a trigger control to the application; updating, by thecomputer, the trigger control based on a movement of a corner of thetrigger control; receiving, by the computer, a selection of a second tabof the application; updating, by the computer, the control image sent tothe display to include the color data and the depth data after thesecond tab is selected; and, associating, by the computer, the triggerarea and the trigger volume with the trigger control.